LLM APPLICATION SECURITY

LLM Application Security

Large Language Models (LLMs), such as OpenAI's GPT-4, Google’s Gemini, and Meta’s LLaMA, have brought significant advancements to natural language processing (NLP) and AI-driven applications. Their ability to generate text, answer questions, and engage in conversations has been transformative for sectors such as customer service, content creation, and education.

Since the launch of ChatGPT, there has been a surge in ChatGPT and LLM-based applications and projects. At the time of this writing, there are close to 300,000 GPT and LLM-related open-source projects on GitHub. As businesses strive to stay ahead of the competition, many are rushing to implement generative AI solutions, often leveraging open-source software to develop their business tools. Software such as LangChain, LlamaIndex, and Ollama are widely used to create generative AI solutions.

However, a report published by Rezilion [1], which investigated the security posture of the 50 most popular generative AI projects on GitHub, found that the majority of them scored very low, with an average of 4.6 out of 10. This is alarming, as some of these projects are widely used by developers to build other LLM applications. Vulnerabilities in these projects can have downstream repercussions throughout the software supply chain. Attackers are already exploiting the rising popularity of this software. For example, a vulnerability reported in LangChain [2] makes it susceptible to prompt injection attacks that can execute arbitrary code. Similarly, LlamaIndex [3], which is widely used to develop Retrieval-Augmented Generation (RAG) applications, was found to allow SQL injection through the Text-to-SQL feature in its NLSQLTableQueryEngine [4]. More recently, a vulnerability [5] was discovered in Ollama, a framework widely used to host private LLMs, allowing attackers to upload a malformed GGUF file (a binary file format optimised for fast loading of language model) to crash the application.

As competition in the LLM space intensifies, LLM providers are adding more features to their models, such as tool use and functional extensions with various plug-ins. With these added capabilities, the security implications of potential attacks become even more significant.

OWASP, known for releasing the OWASP Top 10 Web Application Security Risks, has published the OWASP Top 10 for LLM Applications [6], further underscoring the concerns about the security of LLM-based applications. The 2025 list [7] includes, among other risks, prompt injection, sensitive information disclosure, improper output handling, excessive agency, and system prompt leakage.

Prompt injection has consistently been ranked as the top risk in LLM security. A prompt injection attack occurs when a malicious user manipulates the operation of a trusted LLM by injecting specially crafted input prompts, similar to traditional SQL injection or XSS attacks. The injection can occur directly as user prompt to the model or indirectly through various sources such as websites, emails, documents, or other data that an LLM may access during a user session. Some of these prompts may not even be visible or readable by humans.

The following diagram illustrates how a direct prompt injection can occur:

For instance, an attacker might inject a prompt into a customer support chatbot, instructing it to ignore previous guidelines, query private data stores, and send unauthorised emails, leading to privilege escalation and data breaches.

Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. Malicious prompts embedded in external content can alter the model's behaviour in unintended or harmful ways. Figure 2 depicts a scenario where a malicious prompt is embedded in a news site. When the user asks the LLM to summarise the news, the prompt modifies the LLM's behaviour, causing it to make unauthorised purchases on an e-commerce site instead.

References

  1. Rezilion. (n.d.). Expl[AI]ning the Risk: Exploring the Large Language Models Open-Source Security Landscape. Retrieved July 16, 2023, from https://info.rezilion.com/explaining-the-risk-exploring-the-large-language-models-open-source-security-landscape
  2. NIST. (n.d.). CVE-2023-29374. NVD. https://nvd.nist.gov/vuln/detail/CVE-2023-29374
  3. NIST. (n.d.). CVE-2024-23751. NVD. https://nvd.nist.gov/vuln/detail/cve-2024-23751
  4. NIST. (n.d.). CVE-2024-23751. NVD. https://nvd.nist.gov/vuln/detail/cve-2024-23751
  5. NIST. (n.d.). CVE-2024-39720. NVD. https://nvd.nist.gov/vuln/detail/CVE-2024-39720
  6. OWASP Top 10 for Large Language model Applications, OWASP Foundation. https://owasp.org/www-project-top-10-for-large-language-model-applications/
  7. OWASP Top 10 for LLM Applications 2025, OWASP Foundation. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/


Author

Contact Information: Mar Kheng Kok
School of Information Technology
Nanyang Polytechnic
E-mail: [email protected]

Kheng Kok is a Senior Specialist (AI) & Senior Lecturer at Nanyang Polytechnic (NYP). He has diverse and extensive experience in R&D, software development, teaching and professional training. He has served as principal investigator for several research grant projects in the areas of cloud and cybersecurity. He is heavily involved in AI curriculum development in NYP, sits in the AI Technical Committee and has worked extensively with industry partners in the areas of cloud, cybersecurity and AI.